How do Practitioners, PhD Students and Postdocs in the Social Sciences Assess Topic-specific Recommendations?

In this paper we describe a case study where researchers in the social sciences (n=19) assess topical relevance for controlled search terms, journal names and author names which have been compiled by recommender services. We call these services Search Term Recommender (STR), Journal Name Recommender (JNR) and Author Name Recommender (ANR) in this paper. The researchers in our study (practitioners, PhD students and postdocs) were asked to assess the top n preprocessed recommendations from each recommender for specific research topics which have been named by them in an interview before the experiment. Our results show clearly that the presented search term, journal name and author name recommendations are highly relevant to the researchers topic and can easily be integrated for search in Digital Libraries. The average precision for top ranked recommendations is 0.749 for author names, 0.743 for search terms and 0.728 for journal names. The relevance distribution di↵ers largely across topics and researcher types. Practitioners seem to favor author name recommendations while postdocs have rated author name recommendations the lowest. In the experiment the small postdoc group favors journal name recommendations.